Skip to content

FEAT: Save converters and scorers created through the API (part 12) - #4051

Open
varunj-msft wants to merge 438 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Save-Converters-Scorers
Open

varunj-msft wants to merge 438 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Save-Converters-Scorers

Conversation

@varunj-msft

@varunj-msft varunj-msft commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Description

Part 12 of saving Registry entries across restarts. It builds on part 9 (#4027, merged); part 13 (#4052) builds on it.

Design: 4.1 What gets saved, 4.5 Entries that use other entries. Where every part fits: PR map.

This is one of the changes for saving Registry entries across restarts. It builds on #4027, which is merged.

A converter or scorer created through the API is gone when the backend restarts. With "persist": true, POST /api/converters and POST /api/scorers now save it the way POST /api/targets does (#4026), so restore brings it back at every start.

  • The first commit only moves code. The target save steps go from target_service.py to a shared services/saved_entries.py: turn off the server values an entry would read, refuse what it can't hold or a Temporary entry it uses, write the recipe create-only, register it, and delete the recipe again if registering fails. The owner and not-saveable helpers move to routes/common.py, and TargetOrigin (new in FEAT: Save targets created through the API (part 8) #4026) becomes RegistryEntryOrigin. Targets behave as before.
  • persist on converters and scorers follows the target rules. It needs register and a name matching ^[a-z][a-z0-9_]{0,63}$ (422 otherwise). A refusal is a typed 400 (/errors/not-saveable, with the cause in not_saveable_reason), a taken name is 409, and unreadable storage is 503. The owner is recorded in the saved file as for targets. As with targets, a private converter or scorer can't be saved (422), and a taken converter name is a 409 whether or not it's saved.
  • A saved converter or scorer may only use saved or configuration entries. A reference left out of the request but filled from the configured default values (@apply_defaults) is written into the recipe by name, each entry of a list included, so restore uses the same entries even if nothing sets that default at the next start. A default that is Temporary, or isn't a registered entry, is refused. Other configured defaults apply again from the configuration at every start, as they do for startup entries.
  • A saved converter or scorer can use this server's keys and Azure identity, as the ones from the configuration do (DOC: Clarify framework security boundaries and component reuse #4465), so the Azure Speech converters and the Content Safety, audio and local refusal scorers can be saved. It still can't hold a typed key. This takes out the refusals from FIX: Refuse saved converters and scorers that would use this server's credentials (part 11) #4050, which was closed.
  • A converter to save can't take an uploaded file yet, and is built without file parameters, as restore builds it. A type that needs a file says so. A copy made with source can't be saved yet (422).
  • Saving a scorer finds its class and builds it off the event loop, since finding the class can take over a second.
  • Converters and scorers now have origin (saved, configuration or temporary) on list, get and create, and the startup entries of all three kinds are marked as configuration.
  • DELETE /api/converters/{id} returns 409 for a saved converter, since it would come back from its file at the next start. Temporary and configuration converters are removed as before.
  • The scorer name types is now reserved, as it is for targets and converters, since GET /api/scorers/types would hide a scorer with that name.
  • The backend README and pyrit_conf.md describe saving converters and scorers.

Not covered:

  • Converters that pick randomly without a seed, such as the bijection and policy puppetry ones, come back with a new mapping after a restart, as startup converters do.
  • GET /api/scorers/types doesn't say yet why a scorer type can't be saved; saving one refuses it with the reason.
  • Saving converters from the GUI is FEAT: Save converters from the Registry (part 13) #4052 (part 13). Saving copies, deleting or editing saved entries, and keeping uploaded or Blob files on saved converters come later.

Copilot AI balanced review requested due to automatic review settings November 7, 2026 23:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Saved LocalViolenceClassifierScorer instances can still download models using the server’s cached Hugging Face token or HF_TOKEN, with no save-policy refusal or safeguard disabling authenticated downloads.

4 open issues
What changed in this PR

Extends PyRIT’s saved-target infrastructure to converters and scorers created through the API, allowing eligible registry entries to survive backend restarts.

Changes:

  • Adds optional persistence with shared validation, dependency checks, ownership metadata, and rollback on registration failure.
  • Restores saved entries during startup and reinitialization, reports their origin, and protects them from replacement.
  • Includes target-origin and ownership UI, isolated test storage, regression coverage, and configuration documentation.
File Description
tests/​unit/​setup/​test_targets_initializer.py Tests saved-target collision protection.
tests/​unit/​setup/​test_scorer_initializer.py Tests saved-scorer tagging safeguards.
tests/​unit/​setup/​test_converter_initializer.py Tests saved-converter replacement refusal.
tests/​unit/​setup/​test_configuration_loader.py Tests recipe-storage configuration.
tests/​unit/​registry/​test_instance_registry.py Tests origin metadata and replacement guards.
tests/​unit/​registry/​test_attack_technique_registry.py Tests saved-technique collision handling.
tests/​unit/​backend/​test_scorer_service.py Covers scorer persistence and restoration.
tests/​unit/​backend/​test_scorer_routes.py Covers persistence requests and HTTP errors.
tests/​unit/​backend/​test_scenario_suite_routes.py Tests sanitized storage failures.
tests/​unit/​backend/​test_scenario_suite_execution_service.py Tests saved-entry launch conflict reporting.
tests/​unit/​backend/​test_scenario_run_service.py Tests launch-script replacement safeguards.
tests/​unit/​backend/​test_scenario_preset_routes.py Tests sanitized storage failures.
tests/​unit/​backend/​test_saved_entries.py Tests persistent-reference eligibility.
tests/​unit/​backend/​test_mappers.py Tests target origin and ownership mapping.
tests/​unit/​backend/​test_initializer_service.py Tests unavailable local storage responses.
tests/​unit/​backend/​test_auth_middleware.py Tests signed-in user ID extraction.
tests/​unit/​backend/​test_analytics_benchmark.py Verifies benchmark storage isolation.
tests/​unit/​backend/​conftest.py Isolates saved recipes in backend tests.
tests/​end_to_end/​conftest.py Isolates launched backend recipe storage.
test_support/​analytics_benchmark.py Configures benchmark-specific recipe storage.
pyrit/​setup/​initializers/​tools.py Rejects saved-entry collisions during tool setup.
pyrit/​setup/​initializers/​targets.py Guards target grouping against saved collisions.
pyrit/​setup/​initializers/​scorers.py Guards automatic tagging of saved scorers.
pyrit/​setup/​configuration_loader.py Adds recipe-storage source configuration.
pyrit/​score/​true_false/​local_refusal_classifier_scorer.py Declares the scorer unsuitable for persistence.
pyrit/​score/​true_false/​audio_true_false_scorer.py Declares credential-dependent persistence refusal.
pyrit/​score/​scorer.py Adds scorer persistence-policy metadata.
pyrit/​score/​float_scale/​azure_content_filter_scorer.py Declares server credential fallback.
pyrit/​score/​float_scale/​audio_float_scale_scorer.py Declares credential-dependent persistence refusal.
pyrit/​registry/​registry_recipe_storage.py Implements per-registry JSON recipe storage.
pyrit/​registry/​instance_registry.py Tracks origins, owners, and replacement refusals.
pyrit/​registry/​components/​scorer_registry.py Reserves the types instance name.
pyrit/​registry/​components/​attack_technique_registry.py Adds saved metadata and collision protection.
pyrit/​prompt_target/​websocket_copilot_target.py Declares persistence refusal for server sign-in.
pyrit/​prompt_target/​prompt_shield_target.py Declares credential and destination policies.
pyrit/​prompt_target/​openai/​openai_target.py Declares credential, header, and destination policies.
pyrit/​prompt_target/​litellm_chat_target.py Declares API-key fallback.
pyrit/​prompt_target/​hugging_face/​hugging_face_chat_target.py Declares token fallback and persistence refusal.
pyrit/​prompt_target/​http_target/​http_target.py Refuses persistence of raw HTTP requests.
pyrit/​prompt_target/​hack_a_prompt_target.py Declares session and cookie fallbacks.
pyrit/​prompt_target/​github_copilot_target.py Declares GitHub token fallback.
pyrit/​prompt_target/​common/​prompt_target.py Defines target persistence-policy metadata.
pyrit/​prompt_target/​azure_ml_chat_target.py Declares credential and endpoint policies.
pyrit/​prompt_target/​azure_blob_storage_target.py Declares token and container-URL policies.
pyrit/​prompt_target/​a2a_target.py Declares destination parameters for validation.
pyrit/​models/​catalog/​target.py Adds target origin and ownership fields.
pyrit/​models/​catalog/​scorer.py Adds scorer origin.
pyrit/​models/​catalog/​registry_recipe.py Defines recipes, origins, owners, and restore reports.
pyrit/​models/​catalog/​__init__.py Exports recipe-related models lazily.
pyrit/​converter/​converter.py Adds converter credential-fallback metadata.
pyrit/​converter/​azure_speech_text_to_audio_converter.py Declares Azure Speech credential fallback.
pyrit/​converter/​azure_speech_audio_to_text_converter.py Declares Azure Speech credential fallback.
pyrit/​backend/​services/​scorer_service.py Saves scorers and exposes their origins.
pyrit/​backend/​services/​scenario_suite_execution_service.py Exposes actionable saved-entry launch conflicts.
pyrit/​backend/​services/​scenario_run_service.py Detects launch scripts conflicting with saved entries.
pyrit/​backend/​services/​saved_entries.py Shares validation, saving, registration, and rollback.
pyrit/​backend/​services/​runtime_lifecycle.py Restores recipes and reports startup failures.
pyrit/​backend/​services/​converter_service.py Saves converters and guards saved-entry deletion.
pyrit/​backend/​routes/​targets.py Passes persistence context and maps save errors.
pyrit/​backend/​routes/​scorers.py Adds scorer persistence response handling.
pyrit/​backend/​routes/​scenarios.py Documents saved-entry launch conflicts.
pyrit/​backend/​routes/​scenario_suites.py Sanitizes storage diagnostics and list failures.
pyrit/​backend/​routes/​scenario_presets.py Sanitizes storage diagnostics and list failures.
pyrit/​backend/​routes/​initializers.py Handles local storage failures safely.
pyrit/​backend/​routes/​converters.py Maps persistence and deletion errors.
pyrit/​backend/​routes/​common.py Shares owner extraction and save-refusal responses.
pyrit/​backend/​README.md Documents persistence APIs and restrictions.
pyrit/​backend/​models/​targets.py Adds persistence validation and saveability fields.
pyrit/​backend/​models/​scorers.py Adds validated scorer persistence requests.
pyrit/​backend/​models/​converters.py Adds persistence validation and origin fields.
pyrit/​backend/​middleware/​auth.py Exposes the current user’s object ID.
pyrit/​backend/​mappers/​target_mappers.py Maps target origin and saver information.
pyrit/​backend/​mappers/​converter_mappers.py Maps converter origin.
frontend/​src/​utils/​targetOrigin.ts Defines target-origin display labels.
frontend/​src/​utils/​registryName.ts Generates valid, available registry names.
frontend/​src/​utils/​registryName.test.ts Tests name validation and suggestions.
frontend/​src/​types/​index.ts Adds target persistence and ownership types.
frontend/​src/​test-utils/​targetFixtures.ts Supports origin and ownership fixtures.
frontend/​src/​services/​errors.ts Recognizes typed save refusals.
frontend/​src/​services/​errors.test.ts Tests save-refusal recognition.
frontend/​src/​components/​Config/​TargetTable.tsx Displays origins, savers, and ownership filtering.
frontend/​src/​components/​Config/​TargetTable.test.tsx Tests origin display and ownership filtering.
frontend/​src/​components/​Config/​TargetTable.styles.ts Styles saver captions and filter controls.
frontend/​src/​components/​Config/​TargetOriginBadge.tsx Renders target-origin badges.
frontend/​playwright.config.ts Isolates seeded backend recipe storage.
frontend/​e2e/​registry.spec.ts Tests target saving through the GUI.
frontend/​e2e/​fixtures/​manual_send_backend.py Isolates offline fixture recipe storage.
frontend/​e2e/​config.spec.ts Tests saving and ownership UI behavior.
frontend/​e2e/​_targets.ts Extends target mocks with persistence metadata.
doc/​gui/​0_gui.md Explains saved targets and ownership controls.
doc/​getting_started/​pyrit_conf.md Documents recipe storage and restoration behavior.
.pyrit_conf_example Adds commented recipe-storage configuration.

Review effort: Balanced

Comment thread pyrit/score/scorer.py Resolved
Comment thread pyrit/backend/services/saved_entries.py Resolved
Comment thread pyrit/backend/services/saved_entries.py Resolved
@varunj-msft varunj-msft changed the title FEAT: Save converters and scorers created through the API FEAT: Save converters and scorers created through the API (part 12) Nov 9, 2026
… text

Recipes now share one format version, RECIPE_SCHEMA_VERSION, and a file
without one still reads as version 1. A saved target keeps the auth mode
its create request chose as its own field, so restore can replay that
request; only a target may have one.

Every failure message is escaped and shortened where it's built, so a
stored key with a new line or a terminal escape, or a long version number,
can't change a log or run on. A params value nested too deeply for pydantic
is reported as nested too deeply, and the technique recipe says why it
copies the create request's fields.
Restore still reads, checks, and orders the saved entries, but no longer
builds them itself: restore_async takes the builder, so the backend can
build each entry through the API's own create. The copies of the technique
build, the reserved technique names, and the upload-folder refusal are gone.

A name another entry holds is a name clash; a reserved name, or a technique
name a scenario can't use, is an invalid recipe. Each failure message is
escaped and shortened where it's built, and the tests use small test classes
and a builder that raises instead of OpenAI's error text.
Startup and apply now restore saved entries through the API's create for
each kind: a new backend builder turns each recipe back into its create
request and calls the target, converter, scorer, or technique service. A
target is built as an administrator would build it, with this server's
upload folder.
…restore end to end

The configuration docs now say that the backend rebuilds every saved entry
as an administrator, so write access to the source is administrator
authority, and say where ignored files are listed. The startup log says
when a folder couldn't be read instead of asking to fix or delete files.

New tests restore each kind through its create, check the Copilot token
and upload folder rules, run an apply after a startup restore with the real
registry reset, and check one log record per failure without a SAS token.
… a saveability answer once its request goes away

The private-target test now controls the debounce clock, so a slow run can't check the values chosen before Only me.
Signed-in users may create shared converters and scorers that connect with this server's credentials, so a saved one
isn't refused for that. This takes out the refusals of the Azure Speech converters and the Content Safety, audio and
local refusal scorers. A saved entry still can't hold a typed key.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants